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ABSTRACT • . 

A number of recommendations are set forth with the 
purpose of moving toward greater comparability of research results 
and more efficient accumulation of knowledge. . These recommendations 
include more precise definitions of variables and samples, the 
inclusion of marker variables, and a review of the literature to plan 
alignment of new studies with old* . Each of these recommendations is 
worthy of consideration and balanced implementation. Balanced 
implementation is the key, however. Excessive zeal in forcing any of 
these recommendations into regulations would seriously hinder the 
freedom of the researcher to explore and in the final analysis, be 
counter productive. . (Author/RC) 
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The concept of '^Comparability In Research" has been espoused by two 
Federax research coordinating organizations: The Interagency Panel for Early 
Childhood Research and Development and the Interagency Panel for Research and 
Development on Adolescence • These panels are made up of agencies from five 
departments r^epresented by research administrators having responsibilities 
In the health and social sciences. 

The mandate to the Panels for coordination and facilitation of research 
across the many agencies represented on the Panels led to discussions of how 
this could best be accomplished and what obstacles stood in the way. One 
obstacle immediately recognized was the lack of comparability of results among 
research studies in the same area, making it difficult for findings to be 
additive. 

It is" well understood that research produces contradictory results. 
But even when findings are In general agreement, often because of the lack 
of elements of comparability, generalizabllity is weak. In turn, this tends 
to elicit more research on the same question, which suggests a duplication of : 
effort and expense which could have been avoided. The need for resolution 
on this issue was recognized by the researchers themselves in meetings with 
the Panels. As a result, a set of recommendations has been produced which 
has directed attention toward achieving greater comparability. 

First, let us discuss what is meant by "comparability" and then list the • 
proposed recommendations of how this objective may be accomplished. Comparability 
of research results, implies that the results may be compared or aligned. The \ 
intention is to form a better, more refined idea of the "truth. In an ideal v 

■ . ; V 

situation, research findings, through mechanisms and procedures providing 1 



comparability 9 shDuld yield cummulative results across studies. The sum of 
knowledge in the field should be at least equal to the contributing research 
results, if not greater. 

Meeting as members of the two interagency panels, representatives of some 
25 agencies have worked on the problems of comparability and have come up with 
a number of recommendations* Some of these are at the implementation sta^e; 
while action with respect to others is dependent on the state of the research 
art in certain areas of inquiry. Briefly, these recommendations involve; 
(1) an emphasis for alignment in literature reviews; (2) more precise sample 
definitions;* (3) more complete variable definitions; and (4) inclusion of 
marker variables/measures in the research design. 

Although these recommendations and their implementation have already been 
discussed by other participants in this symposium, we would like to offer 
additional comment. While the concept of comparability is not an unreasonable 
one, it is quite possible that these recommendations become, in certain contexts 
not only unreasonable, but dangerous to the future of creative research. It 
is possible to imagine an over-zealous accounting-accountability type within 
the Federal bureaucracy requiring each element of proceeding studies to be 
faithfully reproduced in new ones — thereby not only re-lnventlng the wheel, 
but making certain that the exact dimensions are retained. On a recommendation 
by recommendation basis, let us look at potentially dangerous implementations. 

Literature Review 

First, the review of the literature with special attention to the com- 
parability aspects is a recommendation with which one cannot quarrel on a super- 
ficial basis. The review of the literature traditionally serves a number of 
functions such as providing a conceptual base for the hypothesis to be tested 



or acquainting the researcher with the latest findings which might have a 
bearing on this anticipated research. One function It does not serve is 
indicating how anticipated research can be linked to the existing research In 
the area. Too often the conceptual base is there, but the means of linking 
the results are missing. As a result, the findings stand In Isolation from 
those of other studies which preceeded. In fact. If taken seriously by the 
research community, this recommmendatlon could result In far fewer "Iterative 
discoveries" wherein old findings are presented only clotherd In newrterms. W6 
believe that If serious thought Is devoted to how anticipated and past research 
^ can be aligned, th^ research Itself will be more productive and Its Impact all 
the more pronounced. 

However, again — taken to extreme, the alignment and linkage function can 
be mistaken for a mandate for endless replication to the detriment of progress. 
The demands of comparability cannot overshadow the need for creativity and 
exploration of new areas by Inslstance on repetition of old mistakes. 

Sample Description 

j A second recommendation Is that samples be fully described to allow for 
cross-sample comparisons. This would allow for greater comparability of 
results. Identification of possible sampling biases, etc. While worthy of . ^ 
support In principle, a number of constraints immediately arise, both economic'/ 
and otherwise, not the least of which is the lack of journal space for pre- ^ 
sentatlon of extensive sample descriptions. However, we feel, that the major: 
problem is the cost in dollars and time of obtaining sample descriptor data. 
Beyond the readily apparent or easily obtained data, such as number in sample, ? 
sex, and age or grade, the dollar cost of obtaining data from records of schools 
hospitals, or other Institutions becomes a factor, ^en more expensive is the; 
cost in effort of both researcher and subject in obtaining data on psychologlda^^^ 



or ecological conditions, to say nothing of the Issues Involved In protecting 
the privacy of the subject and his family. All of these constraints tend Co 
militate against extensive efforts toward "precise" sample description. 

Basic to this recommendation Is the value and selection of the diverse 
data elements which together make possible an alignment of results via a 
"precise" definition. The point Is this: I^at sample characteristics should 
be Included? What Is the trade off between the effort needed to obtain these 
data and their later contribution to comparability? These are some of the 
questions which need to be raised and resolved before this recommendation can be 
fully implemented. However, there should be a middle ground which offers a 

partial resolution of this problem. Perhaps a judicious selection of sample 
characteristics based on the literature will produce greater alignment potential. 
This selection could be especially useful In those situations where there may 
be conflicting results or where the generallzablllty of findings Is limited. 

Variable Deflnlton 

The third recommendation is to Include precise definitions of the variables 
under investigation and the Instruments or methods used to measure them. Too 
often, the variables are described in global tezins which renders it impossible 



precisely will be extremely difficult; in others it can be more easily accomplished. 
For example, the argument over what intelligence Is, as well as how to measure 
it, still rages, along with the various factions. Developing an understanding of 
what constitites the elements of a particular variable is an exacting task at 
best, especiallly when the researcher models after Che Alice in Wonderland 
character and proclaims Chat "this variable represents jiist exactly what I 




to identify just what is being researched. In some cases defining variables 



Intend It to represent." It gets even more confounded In these situations 
where multiple names are given to the same variable or multiple variables given 
the same name* This dilema leads to the dictum that the variable is what the 
instrument measures and therefore the test becotbed the definition. In turn, 
more than anything else, this has been responsible for the request for marker 
variables and measures. 

Marker Variables 

The marker variable is intended to function as a "benchmark" to align 
research results. From one vie^^point, this is an extension of the sample 
description, or as an additional characteristic of the sample (with the 

attendant difficulties mentioned above). The selection of these marker 
variables is envisioned as the responsibility of the investigator who, presumably. 
Is in the best position to know those most appropriate to the question under 
study. It is also ^assumed that this same vantage point should allow the 
appropriate marker measure to be selected. One further step is the developing 
of instrument lists from which the researcher could choose between those most 
widely known or most highly rated. However, this would come perl 
to mandatory inclusion of certain measures. 

The problem is determining the appropriate point at which marker variables 
can contribute to wider knowledge without stifling creativity or needlessly 
dictating more for researchers to follow. | 

The marker variable approach Is also limited by the state of the measurement 
technology. In some areas there are sufficient Instruments of good ' technical ■.. 
quality but a dearth exists in others. But even where a good supply does exist, 
the question becomes which of the myriad of potential marker variables should 
he selected and on what criteria? 



Sunnnary 

In summary, a number of recommendations have been set forth with the 
purpose of moving toward greater comparabiltiy of research results and more 
efficient accumulation of knowledge. These recommendations include more precise 
definitions of variables and samples, the inclusion of marker variables, and 

— . 

a review of the literature to plan alignment of new studies with old. Each 

of these recommendations is worthy of consideration and balanced implemeutation. 

i 

Balanced Implementation is the key, however. Excessive zeal in forcing 
any of these recommendations into regulations would seriously hinder the 
freedom of the researcher to explore and in the final analysis, be counter 
productive. 



